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1.  Introduction/Background 


Micro  aerial  vehicles  (MAVs)  have  been  the  focus  of  many  research  thrusts  in  recent  years  with 
regard  to  autonomous  robotics  used  in  surveillance,  reconnaissance,  and  other  observational 
tasks.  Being  able  to  perform  these  tasks  with  unmanned  robotic  vehicles  would  allow  situational 
and  environmental  awareness  without  putting  any  humans  in  harm’s  way.  Furthermore,  they 
extend  the  usefulness  of  unmanned  robotic  vehicles.  MAVs  can  maneuver  in  complex  spaces  and 
are  less  detectable  than  larger  platforms  (i). 

The  first  step  in  developing  a  MAV  that  can  do  any  of  these  tasks  is  flight  control.  Biological 
research  has  shown  many  sensory  components  of  flight  control  in  insects,  one  of  which  is  using 
visual  data  to  maintain  a  straight  path  without  running  into  objects  (2).  Studies  of  insects’  flight 
response  with  respect  to  their  visual  field  have  found  that  when  an  insect  senses  a  moving 
stimulus,  they  turn  towards  that  stimulus  in  order  to  reduce  the  motion  of  their  visual  field  and 
stabilize  their  orientation  with  respect  to  the  environment  (2).  This  response  can  be  recreated  in  a 
robotic  platform  through  the  computation  and  reduction  of  optic  flow. 

Due  to  the  size  of  small-scale  robotics,  there  are  size,  weight,  and  power  limitations  on  the 
onboard  sensors  that  can  be  implemented  on  the  robotic  platforms.  There  is  a  significant  need  for 
small,  light,  less  power-hungry  sensors  and  sensory  data  processing  algorithms  in  order  to 
control  the  robotic  platform  within  the  constraints  of  the  platform  itself.  Due  to  these  constraints, 
using  a  complex  camera  system  would  be  impossible,  so  some  other  vision  sensor  and 
accompanying  algorithm  must  be  implemented  to  compute  optic  flow.  We  propose  using  an 
externally  developed  and  produced  vision  sensor  by  Centeye  to  develop  an  algorithm  for 
computing  optic  flow,  from  existing  algorithms,  to  be  used  to  control  a  quadrotor.  Such  efforts 
will  benefit  the  various  MAV  research  thrusts  by  providing  a  successful  optic  flow  sensor  and 
data  processing  algorithm  that  can  be  shared  and  applied  to  other  robotic  platforms  to  aid  with 
other  research. 

While  the  overarching  goal  of  this  project  is  to  create  a  working  algorithm  for  the  computation  of 
optic  flow  for  multiple  vision  sensors  to  control  the  flight  of  a  quadrotor,  this  project’s  focus  is 
on  characterizing  the  vision  sensors  themselves  and  developing  an  algorithm  that  is  optimized 
for  the  sensor.  In  initial  testing  of  the  sensor,  we  found  a  clear  tradeoff  between  resolution  of  the 
image  acquired  from  the  sensor  and  rate  of  acquisition,  thus  changing  the  focus  of  the  algorithm 
created  for  the  sensor.  The  algorithm  now  computes  optic  flow  from  Centeye ’s  vision  sensor  for 
varying  image  resolutions  to  ultimately  create  a  dynamic  algorithm  that  creates  an  efficient 
tradeoff  between  frame  rate  and  resolution.  In  doing  this,  we  prove  that  trading  off  between 
frame  rate  and  resolution  is  viable  given  a  dynamic  speed  of  the  vehicle  on  which  the  sensors  are 
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mounted  and  find  the  appropriate  ranges  of  vehicle  speeds  for  such  tradeoffs.  Our  hypothesis  is 
that  the  high  resolutions  will  reliably  detect  low  speeds  but,  due  to  the  slow  frame  rate,  will  not 
be  able  to  refresh  fast  enough  to  accurately  detect  higher  speeds.  Low  resolutions,  on  the  other 
hand,  have  a  frame  rate  fast  enough  to  detect  low  and  high  speeds,  but  the  low  resolutions  will 
likely  cause  a  lot  of  noise  in  the  optic  flow  output,  making  them  inaccurate  at  lower  speeds. 

1.1  Optic  Flow 

Optic  flow  is  an  approximation  of  the  apparent  image  motion  caused  by  the  motion  of  the 
observer  relative  to  the  visual  scene.  Its  computation  is  based  upon  local  derivatives  in  a 
sequence  of  images.  In  two  dimensional  (2-D),  optic  flow  specifies  how  much  each  pixel  of  an 
image  moves  between  adjacent  images,  and  in  three  dimensional  (3-D),  it  specifies  how  much 
each  volume  voxel  (volumetric  pixel)  moves  between  adjacent  volumes  (3). 

The  basis  of  differential  optic  flow  is  the  motion  constraint  equation  whose  derivation  is  shown 
in  Barron  and  Thacker’s  “Tutorial:  Computing  2D  and  3D  Optical  Flow”  (3)  and  in  less  detail 
below.  l(x,  y,  t)  is  a  pixel  at  time  t  and  position  (x,  y).  It  moves  by  Sx  and  dy  in  time  St  to  I(x+  Sx, 
y+  Sy,  t+  St),  as  shown  above  in  figure  1. 1(x,  y,  t)  and  I(x+  Sx,  y+  Sy,  t+  St)  are  the  same  pixel 
and  are  therefore  equal,  giving  rise  to  the  following  equation: 

I(x,y,t)  —  I(x  +  Sx,y  +  Sy,t  +  St)  (1) 


Figure  1 .  Optic  flow  illustration. 

The  assumption  that  the  images  are  the  same  is  true  to  a  first  approximation  if  Sx,  Sy,  and  St  are 
not  too  large.  A  first-order  Taylor  series  expansion  about  I(x,  y,  t)  is  performed  to  define  I(x+  Sx, 
y+  Sy,  t+  St). 

I (x  +  Sx,y  +  Sy,  t  +  St)  —  I(x,y,  t)  +  ^-Sx  +  ^-8y  +  ^-8t  +  ■■■  (2) 

ox  dy  ot 
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The  higher-order  Taylor  terms  are  small  enough  to  discard.  Since  I(x,  y,  t)  and  I(x+  Sx,  y+  Sy,  t+ 
St)  are  equal  they  cancel  each  other  out  in  equation  2.  We  can  then  take  the  remaining  equation 
and  divide  by  St. 


dl  dl  dl 

—  Sx  +  —  Sy  +  —St  =  0 

ox  dy  dt 


dl  Sx  dl  Sy  dl  St 
dx  St  +  dy  St  +  dtSt 


(3) 


The  partial  derivatives  dl/dx,  dl/dy,  and  d I/dt  can  be  written  as  Ix,  Iy,  and  It,  the  intensity 
derivatives.  Sx/St  and  Sy/St  can  be  written  as  vx  and  vy,  the  x  and  y  components  of  optic  flow. 
These  substitutions  give  rise  to  the  following  equations. 


Ix^x  T  lyVy  +  It  0 


V/  ■  v  =  -It 


(4) 


The  final  equation  is  the  2-D  Motion  Constraint  Equation,  where  V/  =  (jx,  7y)is  the  spatial 
intensity  gradient  and  v  =  (vx,  vy)  is  the  image  velocity  or  optic  flow  at  pixel  (x,  y )  at  time  t. 


Optic  flow  algorithms  calculate  the  image  velocity  or  optic  flow  vector  v  shown  in  the  2-D 
Motion  Constraint  Equation  (equation  4).  Optic  flow  can  be  computed  by  feature  tracking 
(tracking  the  motion  of  corresponding  features  in  adjacent  images),  comparing  local  gradients  of 
image  intensity  in  space  and  time,  or  comparing  the  temporal  frequencies  in  various  regions  of 
the  spatiotemporal  frequency  spectrum  (5).  Optic  flow  algorithms  have  been  developed  for  each 
of  these  approaches  and  the  two  that  are  tested  using  Centeye’s  vision  sensor  are  discussed. 


1.2  Lucas  Kanade  Algorithm 

The  Lucas-Kanade  (LK)  method  is  a  differential  method  for  optical  flow  estimation  developed 
by  Bruce  D.  Lucas  and  Takeo  Kanade,  which  assumes  a  constant  model  for  v  in  a  small  spatial 
neighborhood  and  solves  the  Motion  Constraint  Equation  for  all  the  pixels  in  that  neighborhood 
by  implementing  a  local  least-squares  approximation  ( 4 ).  LK  takes  the  2-D  Motion  Constraint 
Equation  and  converts  it  into  matrix  form.  That  matrix  form  is  shown  as  follows: 

(V/)v  =  -It  ^ 


This  equation  is  then  solved  using  least-squares  approximation. 
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(V/f(V/)  17  =  (V/)r(— /t) 
v  =  {(V/)T(V/)}-1(V/)T(-/t) 


(6) 


This  method  of  computing  optic  flow  has  also  been  expanded  upon  and  improved  through  the 
use  of  pyramid  interpolation  and  hierarchical  methods  for  more  precise  object  tracking.  The  LK 
algorithm  is  one  of  the  most  commonly  used  algorithms  for  computing  optic  flow  in  computer 
vision  due  to  its  assumption  that  the  optical  flow  in  a  small  neighborhood  of  pixels  is  the  same, 
allowing  for  a  simplification  of  a  least- squares  approximation  (9). 

1.3  Image  Interpolation  Algorithm 

Srinivasan  proposes  an  algorithm  (the  image  interpolation  algorithm  [IIA])  for  computing  optic 
flow,  which  does  not  require  the  tracking  of  features  or  measurement  of  image  velocities  at  many 
different  locations.  Rather  IIA  estimates  optic  flow  by  a  non-iterative  process,  which  interpolates 
the  position  of  the  moving  image  with  respect  to  a  set  of  reference  images  (<5).  IIA  estimates  the 
motion  of  a  plane  that  is  capable  of  translation  along  the  x-  and  y-axes,  which  are  perpendicular 
to  the  camera  axis,  and  rotate  about  an  arbitrary  axis  perpendicular  to  the  plane.  The  motion 
estimate  of  the  plane  is  done  within  a  patch  of  the  plane,  which  is  defined  by  a  window  function. 
The  following  equations  used  for  the  computation  of  optic  flow  in  IIA  come  from  Srinivasan’s 
“An  Image-Interpolation  Technique  for  the  Computation  of  Optic  Flow  and  Egomotion”  (6). 

Intensity  functions  of  two  images  in  time  from  to  to  t  are  denoted  by  fo  (x,y,to)  and  f(x,y,t).  IIA 
interpolates /from/o  and  the  references  images//,  Z?...  fo-  These  reference  images  are  defined  as 
versions  of/i/by  shifts  in  the  x,  y,  and  9  directions. 

fi(x,  y)  =  f0(x  +  Axre/) 

/20>y)  =  fo(x  -  Axref) 

/sO.  y)  =  /o(y  +  Ayre/) 

/4(x,y)  =/0(y-  Ay ref) 
fs(x,y)  =/0(V,y') 

ZeO-y)  =/o(*".y")  (7) 

where  the  x’,x”,y  ’,  and  y  ’  ’  expressions  are  as  follows: 
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x'  =  x  ■  sinA9ref  +  y  ■  cosA9ref 
x"  —  x  ■  sin(—A9ref)  +  y  ■  cos{—A9ref) 
y'  —  x  •  cosA9ref  +  y  ■  sinA9ref 

y"  —  x  ■  cos(—A9ref)  +  y  ■  sin(—A9ref)  (8) 

We  assume  that/can  be  approximated  by/: 

f  =  A  +  as (£?) ~ «  +  °'5 (£7)  W  - «  +  °'5 (£7)  &  (9) 

This  assumption  is  true  if  the  displacement  of  the  reference  images  fuf2--f6  and  image /relative 
to  the  original  image  fo  are  small  compared  to  the  highest  spatial-frequency  component’s  spatial 
wavelength  in  the  image.  IIA  is  not  as  computationally  complex  as  LK,  but  rather  takes  its 
inspiration  from  biology  to  linear  combine  shifted  versions  of  the  original  image  captured  to 
compute  optic  flow. 

1.4  Centeye’s  ArduEye  Vision  Sensor 

Centeye  is  a  corporation  started  by  Geoffrey  Barrows,  which  specializes  in  embedded  vision 
systems  for  robotics  applications  (7).  Centeye  develops  software  and  hardware  for  ArduEye, 
which  is  an  open  source  project  for  the  implementation  of  vision  sensors  using  an  Arduino 
microcontroller  (5).  These  vision  sensors  or  “vision  chips”  are  similar  to  regular  charge-coupled 
device  (CCD)  or  complementary  metal-oxide  semiconductor  (CMOS)  imaging  chips,  which 
convert  an  optical  image  into  electrical  signals  but  unlike  the  CCD  and  CMOS  chips,  which 
digitize  the  image,  a  vision  chip  performs  image  processing  on  the  raw  image  using  a  mixture  of 
analog  and  digital  circuitry  (7). 

The  output  of  the  vision  chip  is  a  pre-processed  version  of  the  image;  therefore,  the  resolution  of 
the  image  can  be  reduced  during  acquisition  as  opposed  to  performing  post-acquisition  reduction 
on  the  Arduino,  saving  processing  time  and  memory.  This  pre-processing  property  of  the 
ArduEye  sensor  was  exploited  so  that  we  could  specify  the  number  of  pixels  binned  in  order  to 
decrease  the  resolution  of  the  image  acquired.  In  this  way,  only  every  2,  4,  8,  or  16  pixels  were 
acquired  to  represent  the  “super  pixel”  of  the  2x2,  4x4,  8x8,  or  16x16  sections.  The  resolutions 
of  the  acquired  image  after  this  pre-processing  are  56x56,  28x28,  14x14,  and  7x7,  respectively. 

The  ArduEye  chip  used  in  this  project  is  from  the  Stonyman  series,  which  was  connected  to  the 
Arduino  Mega  2560  board  using  the  ArduEye  Rocket  System  breakout  board,  all  of  these 
components  are  shown  in  figure  2.  Two  Stonyman  chips  were  initially  tested  to  determine  which 
one  would  be  used  for  the  project.  One  chip  uses  pinhole  optics,  which  finds  the  brightest  pixel 
in  the  image  and  acquires  a  16x16  image  centered  at  that  pixel,  and  the  other  chip  uses  cell 
phone  optics,  which  can  acquire  up  to  a  112x112  image.  Both  of  these  chips  are  shown  in 
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figure  2;  the  cell  phone  optics  chip  is  on  top  and  pinhole  optics  chip  is  on  the  bottom.  Because 
the  eventual  goal  of  this  project  is  flight  control  of  a  quadrotor,  the  cell  phone  optics  chip  was 
used  to  acquire  a  larger  image  for  more  precise  optic  flow  measurements. 


Figure  2.  ArduEye  vision  chip  on  Stonyman  breakout  board  connected  to  Arduino 
Mega  (8)  (left)  and  the  Stonyman  vision  chips  (7)  (right). 

Much  of  the  code  used  in  this  project,  such  as  image  acquisition  and  optic  flow  computation,  was 
written  by  Centeye  and  put  out  as  open  source  downloadable  code  from  their  Web  site  (8).  Since 
all  of  this  code  is  available,  it  is  not  specifically  discussed  in  this  report,  but  rather  is  mentioned 
with  the  description  of  the  optic  flow  algorithm  in  section  2. 


2.  Experiment/Calculations 


The  optic  flow  algorithm  implemented  is  a  simple  loop  that  acquires  the  current  image  from  the 
vision  sensor  and  then  uses  that  image  and  the  previously  acquired  image  to  compute  optic  flow. 
That  high  level  loop  is  shown  in  figure  3.  The  variability  in  the  algorithm  comes  with  the  bin 
number,  which  determines  the  resolution  of  the  image  acquired  from  the  chip  and  the  switch 
between  using  LK  or  IIA  to  compute  optic  flow.  This  variability  in  the  algorithm  is  illustrated  in 
figure  3  as  variables  that  are  passed  in  to  the  functions  within  the  high  level  task  pictured.  All  the 
functions  used  to  acquire  the  image  from  the  sensor  and  compute  optic  flow  are  taken  from  the 
ArduEye  Web  site  (8).  The  only  additions  to  the  code  are  the  implementation  of  those  functions 
in  a  simple  loop  and  the  printing  of  the  data  to  conform  to  the  LabView  data  acquisition  system 
used  for  the  experiments  conducted  on  the  rate  table.  The  LK  and  IIA  functions  were  written  by 
Centeye  and  implement  the  equations  presented  in  section  1  using  pointers;  those  functions  can 
be  found  on  the  ArduEye  Web  site  (§).  Additional  functions  were  written  for  initial  testing  but 
were  not  used  in  the  final  algorithm.  Those  functions  are  discussed  with  the  first  experiment. 
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Figure  3.  Optic  flow  algorithmic  loop. 

Two  experiments  were  conducted  within  this  project  to  provide  initial  characterization  data  of 
the  sensors  and  their  performance.  The  first  experiment  attains  the  Arduino’s  processing  times 
needed  to  acquire  the  image  from  the  sensor  and  compute  optic  flow  for  each  image  resolution 
previously  discussed.  This  initial  timing  experiment  is  conducted  to  determine  the  performance 
of  the  Arduino,  which  dictates  the  final  algorithm.  The  second  experiment  determines  the 
accuracy  of  optic  flow  computations  for  each  image  resolution  at  varying  speeds  by  performing 
rate  table  experiments.  The  specifics  of  these  experiments  are  discussed  in  sections  2.1  and  2.2. 

2.1  Timing  Experiment 

This  experiment  uses  only  the  Arduino  software  and  the  ArduEye  vision  sensor  to  determine  the 
processing  time  of  functions  used  in  the  optic  flow  algorithm.  These  times  are  acquired  using  the 
Arduino  function  millis(),  which  returns  number  of  milliseconds  since  the  Arduino  began 
running  the  current  program.  This  time  is  reported  after  each  run  of  the  main  loop  in  the 
program,  and  thus  the  timing  data  are  taken  as  an  average  of  the  difference  between  the  times 
reported  by  millis()  on  the  serial  monitor. 

2.2  Rate  Table  Experiment 

The  rate  table  experiment  uses  the  optic  flow  algorithm  to  estimate  angular  velocity.  The 
ArduEye  sensor  is  mounted  on  a  rate  table  and  spun  at  various  rates  to  characterize  the  noise  and 
sensitivity  the  sensor  at  each  resolution  using  both  LK  and  IIA.  The  algorithm’s  estimate  of 
angular  velocity  is  compared  with  the  actual  rate  at  which  the  table  is  being  driven  to  determine 
noise  and  sensitivity.  Since  translation  velocity  is  held  to  zero  with  the  sensor  mounted  at  the 
center  of  the  rate  table,  optic  flow  is  taken  as  a  direct  correlation  to  angular  velocity.  This  can  be 
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realized  through  equation  10  (1)  in  which  OF  is  optic  flow,  co  is  angular  velocity,  v  is 
translational  velocity,  and  D  is  the  distance  of  the  sensor  to  an  object. 

OF  =  -oi  +  ^  (10) 

When  translational  velocity  v  is  zero,  optic  flow  is  proportional  to  angular  velocity  co.  In  this 
way,  the  rate  table  experiment  is  a  simple  way  to  reduce  the  number  of  variables  in  the  optic 
flow  equation  10  and  characterize  the  noise  and  sensitivity  of  the  sensor. 

Figure  4  shows  a  picture  of  the  experimental  setup.  For  a  rate  table,  we  used  a  motor  mounted 
under  a  test  stand,  which  rotates  the  metal  bar  attached  to  the  stand.  We  then  attached  the 
Arduino  board  with  the  ArduEye  sensor  to  the  bar  and  read  data  off  of  the  Arduino  to  a  computer 
running  LabView.  The  LabView  program  then  compiled  a  text  file  of  the  optic  flow  values  sent 
by  the  Arduino  and  the  time  stamp  at  which  those  values  were  reported.  The  speed  of  the  motor 
was  controlled  through  a  laptop,  which  told  the  motor  when  to  start  and  stop  and  at  what  speed  to 
rotate. 


Figure  4.  Rate  table  experimental  setup. 
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3.  Results  and  Discussion 


3.1  Timing  Experiment 

In  initial  testing  of  the  optic  flow  algorithm,  we  found  that  the  Arduino  Mega  board  could  not 
acquire  and  save  the  112x112  image  in  its  flash  memory  and  therefore  another  function  was 
written  to  acquire  and  save  the  image  from  the  vision  sensor  to  the  EEPROM  on  the  board. 
Another  hardware  constraint  found  was  that  after  acquiring  an  image  and  saving  it  either  to  flash 
memory  or  EEPROM,  resolutions  higher  than  28x28  could  not  be  passed  into  the  optic  flow 
functions  (both  LK  and  IIA)  without  overflowing  the  Arduino  flash  memory  and  causing  the 
entire  program  to  crash.  To  deal  with  this  constraint,  we  downsized  the  acquired  image  by 
averaging  across  2x2  and  4x4  sections  to  acquire  averaged  “super  pixels,”  as  opposed  to  the 
binned  “super  pixels”  acquired  originally,  which  sample  one  pixel  to  represent  a  group  of  pixels. 
Both  the  EEPROM  and  averaged  “super  pixel”  added  functions  became  part  of  the  timing 
experiment. 

Figure  5  shows  the  results  of  the  timing  experiment  for  all  five  discussed  resolutions  with  the 
following  four  functions:  acquiring  and  saving  the  image  from  the  vision  sensor  to  the 
EEPROM,  saving  the  image  to  the  flash  memory,  performed  the  LK  optic  flow  computation,  and 
performing  the  IIA  computation.  The  LK  and  IIA  processing  times  overlay  one  another  on  the 
graph. 

The  first  immediate  result  we  can  see  from  this  graph  is  that  by  increasing  the  resolution  by  a 
window  size  factor  of  2  each  function’s  time  increases  by  a  factor  of  around  4,  which  is  to  be 
expected  given  that  an  increase  by  2  in  the  window  size  is  a  increase  by  4  in  the  amount  of  pixels 
in  the  image.  We  can  also  see  that  although  saving  to  the  EEPROM  takes  significantly  more  time 
than  saving  to  the  flash  memory  (an  increase  by  a  factor  of  between  8  and  23),  saving  to  the 
EEPROM  is  the  only  option  for  acquiring  the  full  112x112  resolution.  Similarly,  IIA  and  LK  can 
only  be  computed  for  7x7,  14x14,  and  28x28  resolutions. 
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The  results  of  the  timing  experiment  are  shown  in  figure  6  for  the  averaging  “super  pixel” 
solution  to  the  size  resolution  constraints  of  the  optic  flow  functions.  We  can  see  from  this  graph 
that  even  with  4x4  pixel  averaging,  the  greatest  resolution  attainable  is  56x56,  so  the  entire 
112x112  resolution  cannot  be  downsized  in  order  to  have  optic  flow  computed.  This  information 
was  useful  to  identify  the  limitations  of  the  Arduino  microcontroller. 
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OF  Algorithm  with  Superpixel  Reduction 
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-*-2x2 

-*-4x4 


Figure  6.  “Super  pixel”  reduction  processing  times. 

The  conclusion  drawn  from  these  timing  experiments  is  that  the  Arduino’s  memory  cannot 
handle  higher  resolutions  without  reduction  using  “super  pixels,”  and  even  with  that  fix,  the 
highest  resolutions  cannot  be  processed.  In  order  to  move  forward  with  this  project  and  use  the 
ArduEye  vision  sensors  for  flight  control  of  a  quadrotor  we  need  to  move  toward  a 
microcontroller  with  more  memory  and  processing  power. 

3.2  Rate  Table  Experiment 

The  results  of  the  timing  experiments  restricted  the  image  resolutions  that  could  be  tested  on  the 
rate  table  to  7x7,  14x14,  and  28x28  pixels.  Each  resolution  was  run  at  six  different  speeds,  both 
positive  and  negative,  and  at  rest.  Each  speed  was  maintained  for  a  variable  amount  of  time, 
speeds  were  switched  either  once  we  decided  enough  data  were  collected  or  if  the  speed  could 
not  be  maintained  anymore  due  to  physical  constraints  such  a  the  input/output  (I/O)  wire  over 
twisting.  These  tests  were  done  for  both  the  LK  and  IIA  algorithm  to  determine  the  differences  in 
the  sensor’s  noise  and  sensitivity  between  each  of  those  algorithms. 
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Figures  7  and  8  show  the  raw  results  for  both  LK  and  IIA  at  all  resolutions  and  speeds.  These 
speeds  are  slightly  different  between  the  LK  and  IIA  tests  because  the  motor  speed  controller 
proved  to  be  slightly  variable  and  exact  same  speeds  could  not  be  accomplished.  These  graphs 
show  the  optic  flow  values  as  computed  by  the  Arduino.  The  x-axis  represents  time  but  as  the 
indices  of  the  optic  flow  data  in  an  array,  not  as  the  actual  time  stamp  of  the  time  at  which  the 
optic  flow  value  was  recorded.  The  rate  at  which  the  table  was  rotating  in  radians  per  second  is 
noted  above  or  below  the  section  of  the  graph  to  which  it  corresponds.  The  six  speeds  were 
broken  into  two  continuous  tests  in  which  three  speeds  (both  positive  and  negative)  were  tested 
in  succession. 


Figure  7.  IIA  rate  table  raw  data. 
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Figure  8.  LK  rate  table  raw  data. 

The  first  conclusion  drawn  from  this  data  was  that  the  rate  of  image  acquisition  for  each 
resolution  agreed  with  the  times  attained  in  the  timing  experiment.  These  times  were  attained 
from  the  raw  data  by  averaging  the  difference  in  the  times  that  the  optic  flow  was  obtained. 

Thus,  the  frame  rates  of  the  7x7,  14x14,  and  28x28  resolutions  were  confirmed  to  be  ~45,  —18, 
and  ~6  Hz,  respectively. 

To  analyze  these  raw  data,  the  mean  of  the  optic  flow  for  each  rate  was  computed  as  well  as  the 
standard  deviation.  The  means  and  standard  deviations  were  taken  across  a  window  of  the 
speed’s  section  of  data  such  that  the  same  number  of  data  points  were  being  used  to  calculate  the 
mean  and  standard  deviation  for  each  speed  and  resolution.  The  number  of  data  points  in  each 
window  was  determined  by  the  smallest  window  that  number  of  points  was  150.  These  data  were 
then  scaled  to  reflect  the  actual  rate  at  which  the  table  was  being  driven  to  yield  the  graphs 
shown  in  figures  9  and  10.  The  means  are  graphed  as  a  function  of  the  rate  at  which  the  table  was 
being  driven  and  the  standard  deviations  are  shown  as  the  error  bars  associated  with  each  optic 
flow  mean. 
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Figure  9.  Lucas-Kanade  (LK)  Optic  flow  vs.  speed,  with  low  speeds  data  points  shown  in  blow  up  graph. 


graph. 

The  7x7  optic  flow  for  both  LK  and  IIA  exhibits  a  fairly  linear  trend  with  the  rate  at  which  the 
sensor  is  being  driven,  demonstrating  that  the  low  resolution  has  a  large  range  of  high  sensitivity. 
However,  a  lot  of  noise  is  seen  within  that  entire  range.  The  higher  resolutions  of  14x14  and 
28x28  have  significantly  less  noise  but  are  only  high  sensitivity  at  low  speeds.  The  graphs  of  the 
blown  up  lower  speeds  show  that  all  resolutions  tested  are  accurate  for  speeds  less  in  magnitude 
than  ~2  radians  per  second,  but  outside  that  range  28x28  drops  off  significantly  in  sensitivity. 
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The  14x14  resolution  sees  a  similar  drop  off  in  sensitivity  in  speeds  greater  in  magnitude  than 
-10  radians  per  second.  These  results  confirm  our  hypothesis  that  as  the  rate  at  which  the  sensor 
is  being  driven  increases,  the  higher  resolutions’  slower  frame  rate  cannot  capture  images  often 
enough  to  get  an  accurate  estimate  of  optic  flow.  Although  the  7x7  frame  rate  can  acquire  images 
fast  enough,  the  lower  resolution  introduces  a  lot  of  noise  into  the  system  due  to  the  smaller 
number  of  pixels  used  to  represent  the  entire  image.  The  higher  resolutions  do  not  introduce  as 
much  noise  due  to  the  greater  number  of  pixels  giving  a  better  view  of  the  actual  image  that  the 
vision  sensor  sees. 

The  outlier  in  these  results  is  the  data  from  rest.  For  all  resolutions,  the  zero  speed  has  higher 
noise  then  the  neighboring  low  speeds  and  for  7x7  LK  that  zero  value  appears  as  a  value  of  ~2 
radians  per  second  whereas  the  other  resolutions  yield  a  value  much  closer  to  0  radians  per 
second.  This  bias  in  the  LK  7x7  data  may  be  due  to  the  way  in  which  data  were  collected  and 
where  the  vision  sensor  was  pointing  when  the  zero  data  were  collected.  Optic  flow  is  a  measure 
of  image  velocity  so  if  while  the  sensor  was  at  rest  its  image  field  changed,  that  would  read  as 
non-zero  optic  flow.  The  reasoning  could  also  explain  the  high  noise  associated  with  the  zero 
data.  The  change  in  the  image  field  due  to  human  movement  or  light  changes  can  likely  be 
neglected  when  the  sensor  is  being  driven  at  a  non-zero  rate.  This  is  because  the  rate  at  which  the 
sensor  is  moving  has  more  of  an  impact  on  the  optic  flow  output  than  the  ambient  movement  of 
the  environment.  This,  however,  is  not  true  at  rest  and  may  account  for  the  bias  seen  in  that  data. 

The  conclusion  we  can  draw  from  these  data  is  that  there  is  a  clear  tradeoff  between  frame  rate 
and  resolution,  which  can  be  exploited  to  optimize  sensitivity  and  noise  relative  to  the  speed  at 
which  the  sensor  is  moving. 


4.  Conclusions 


The  results  of  the  timing  experiments  revealed  that  the  Arduino’s  memory  cannot  handle  higher 
resolutions  higher  than  28x28  pixels;  therefore,  we  need  to  move  towards  a  microcontroller  with 
more  memory  and  processing  power.  The  next  step  in  this  project  will  be  to  test  the  ArduEye 
sensors  on  a  processor  such  as  the  ARM.  The  same  characterization  experiments  will  take  place 
with  that  board. 

The  rate  table  results  proved  that  the  low  resolution  image  has  a  large  range  of  high  sensitivity 
for  optic  flow,  but  a  lot  of  noise  is  seen  within  that  entire  range.  Higher  resolutions  have 
significantly  less  noise  but  are  only  highly  sensitivity  at  low  speeds.  These  results  were  as 
expected  and  the  same  results  should  come  when  the  ArduEye  sensor  is  tested  with  the  ARM 
processor.  If  frame  rate  can  be  sped  up  for  the  higher  resolutions  using  this  processor,  these 
results  can  be  improved  such  that  the  higher  resolutions  have  a  higher  sensitivity  over  a  greater 
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range  of  speeds.  Also,  with  a  processor  with  greater  memory,  the  even  higher  resolutions  of 
56x56  and  1 12x1 12  pixels  can  be  tested. 

This  research  proved  that  there  is  a  clear  tradeoff  between  frame  rate  and  resolution,  leading  to 
either  more  sensitive  results  with  high  noise  or  less  sensitivity  with  lower  noise.  We  proved  that 
at  low  speeds  the  higher  resolutions  of  28x28  and  14x14  pixels  perform  as  well  as  the  7x7  low 
resolution  but  with  less  noise,  whereas  the  sensitivities  of  those  high  resolutions  drop  off 
immediately  after  low  speeds.  Those  drop-off  speeds  were  determined  to  be  ~2  radians  per 
second  in  magnitude  for  28x28  and  ~10  radians  per  second  in  magnitude  for  14x14.  In  this  way, 
the  high  resolutions  of  14x14  and  28x28  are  just  as  accurate  and  more  reliable  for  the  lower 
speeds  (~10  radians  per  second),  but  the  7x7  low  resolution  must  be  used  for  any  speed  outside 
of  that  range.  Using  these  results  as  a  proof  of  concept,  a  dynamic  algorithm  can  be  created  that 
sets  the  resolution  of  the  image  being  acquired  according  to  the  speed  of  the  vehicle.  More  tests 
need  to  be  performed  before  such  an  algorithm  is  created  to  test  the  sensitivity  and  noise  of  the 
sensor  at  varying  translational  velocities  in  additional  to  purely  angular  velocities. 

The  overall  goal  of  this  project  is  to  integrate  multiple  ArduEye  sensors  on  one  platform  for  fully 
autonomous  flight  control,  so  future  work  needs  to  focus  on  how  to  combine  the  optic  flow 
outputs  of  multiple  sensors  to  translate  those  combined  outputs  into  actuation.  Flight  control 
using  optic  flow  must  also  take  in  account  the  dynamics  of  the  vehicle  being  flown  as  well  as  the 
environment  in  which  the  flight  is  being  attempted.  All  subsequent  work  will  build  off  of  the 
vision  sensor  characterization  results  from  this  project. 


16 


5.  References 


1.  Green,  W.  E.;  Oh,  P.  Y.  Optic-Flow-Based  Collision  Avoidance.  IEEE  Robotics  & 
Automation  Magazine  2008,  96-103. 

2.  Srinivasan,  M.  V.;  Poteser,  M.  Motion  Detection  in  Insect  Orientation  and  Navigation.  Vis. 
Res.  1999,  29,  2749-103. 

3.  Barron,  J.  L.;  Thacker,  N.  A.  Tutorial:  Computing  2D  and  3D  Optical  Flow,  Tina  Memo  No. 
2004-012;  Manchester,  M13  9PT,  2005. 

4.  Lucas,  B.  D.;  Kanade,  T.  An  Iterative  Image  Registration  Technique  with  an  Application  to 
Stereo  Vision.  DARPA  Image  Understanding  Workshop  1981,  121—130. 

5.  University  of  Central  Florida  Department  of  EECS  Computer  Science  Division.  Lecture  17 
Computing  Optical  Flow:  Lucas  &  Kanade. 

http://www.cs.ucf.edu/courses/cap6411/cap5415/Lecture-16.PDF  (accessed  July  2012). 

6.  Srinivasan,  M.  V.  An  Image-Interpolation  Technique  for  the  Computation  of  Optic  Flow 
and  Egomotion  Biol.  Cybern.  1994,  71,  401-415. 

7.  Centeye,  Inc.  http://centeye.com  (accessed  June  2012). 

8.  ArduEye:  An  open  source  programmable  vision  sensor  for  Arduino. 
http ://www. ardueye .com  (accessed  June  2012). 

9.  Duhamel,  P.  E.;  Poter,  J.;  Finio,  B.;  Barrow,  G.;  Brooks,  D.;  Wei,  G.  Y.;  Wood,  R. 

Hardware  in  the  Loop  for  Optical  Flow  Sensing  in  a  Robotic  Bee.  IEEE/RSJ  International 
Conference  on  Intelligent  Robots  and  Systems  2011,  1099-1106. 


17 


List  of  Symbols,  Abbreviations,  and  Acronyms 


2-D 

two  dimensional 

3-D 

three  dimensional 

CCD 

charge-coupled  device 

CMOS 

complementary  metal-oxide  semiconductor 

I/O 

input/output 

IIA 

image  interpolation  algorithm 

LK 

Lucas-Kanade 

MAVs 

micro  aerial  vehicles 
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